Add compaction stats for filtered files #13136
Conversation
|
@jowlyzhang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
|
@jowlyzhang has updated the pull request. You must reimport the pull request before landing. |
|
@jowlyzhang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
| @@ -30,6 +32,7 @@ void CompactionJobStats::Reset() { | |||
| total_blob_bytes_read = 0; | |||
| total_output_bytes = 0; | |||
| total_output_bytes_blob = 0; | |||
| total_skipped_input_bytes = 0; | |||
There was a problem hiding this comment.
nit: total_filtered_input_bytes to be consistent with the other new stats
There was a problem hiding this comment.
Yeah, thanks for the suggestion! I thought a bit about this and feel the action of filtering is applied to the file, and it ends up getting all the bytes in the file skipped, as opposed to the bytes are filtered. So I made this distinction in the name. The documentation of the field should make it clear where these fields come from.
|
@cbi42 Thanks for the review! |
|
@jowlyzhang merged this pull request in ef119c9. |
As titled. This PR adds some compaction job stats, internal stats and some logging for filtered files.
Example logging:
[default] compacted to: files[0 0 0 0 2 0 0] max score 0.25, estimated pending compaction bytes 0, MB/sec: 0.3 rd, 0.2 wr, level 6, files in(1, 0) filtered(0, 2) out(1 +0 blob) MB in(0.0, 0.0 +0.0 blob) filtered(0.0, 0.0) out(0.0 +0.0 blob), read-write-amplify(2.0) write-amplify(1.0) OK, records in: 1, records dropped: 1 output_compression: Snappy
Test plan:
Added unit tests